Breakthroughs, discoveries, and DIY tips sent every weekday. It's something out of a dream or TV show: a married couple takes their dog for a walk and finds a buried treasure worth $10 million. But it actually happened, back in 2013. The treasure is the Saddle Ridge Hoard, the largest ever stash of gold coins found in the United States. The couple, who go by John and Mary in the press, have been careful to obscure their identity and the exact place where they live to prevent would-be treasure hunters from showing up on their property.

With the introduction of (large) language models, there has been significant concern about the unintended bias such models may inherit from their training data. A number of studies have shown that such models propagate gender stereotypes, as well as geographical and racial bias, among other biases. While existing works tackle this issue by preprocessing data and debiasing embeddings, the proposed methods require a lot of computational resources and annotation effort while being limited to certain types of biases. To address these issues, we introduce REFINE-LM, a debiasing method that uses reinforcement learning to handle different types of biases without any fine-tuning. By training a simple model on top of the word probability distribution of a LM, our bias agnostic reinforcement learning method enables model debiasing without human annotations or significant computational resources. Experiments conducted on a wide range of models, including several LMs, show that our method (i) significantly reduces stereotypical biases while preserving LMs performance; (ii) is applicable to different types of biases, generalizing across contexts such as gender, ethnicity, religion, and nationality-based biases; and (iii) it is not expensive to train.